<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html
  PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xml:lang="zh" lang="zh">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>
<meta name="DC.Type" content="concept"/>
<meta name="DC.Title" content="如何创建索引"/>
<meta name="DC.subject" content="Apache, Lucene, 原理, 代码, OSSEZ, OSS 计算技术"/>
<meta name="keywords" content="Apache, Lucene, 原理, 代码, OSSEZ, OSS 计算技术"/>
<meta name="DC.Relation" scheme="URI" content="../../tasks/chapter01/chapter01.html"/>
<meta name="prodname" content="Lucene 指南"/>
<meta name="version" content="1.0.0.0"/>
<meta name="brand" content="OSSEZ APACHE LUCENE"/>
<meta name="copyright" content="OSSEZ (中国) 信息技术有限公司 2006, 2012" type="primary"/>
<meta name="DC.Rights.Owner" content="OSSEZ (中国) 信息技术有限公司 2006, 2012" type="primary"/>
<meta name="DC.Format" content="XHTML"/>
<meta name="DC.Identifier" content="chapter01"/>
<meta name="DC.Language" content="zh"/>
<link rel="stylesheet" type="text/css" href="../../css/commonltr.css"/>
<link rel="stylesheet" type="text/css" href="../../css/public.css"/>
<title>如何创建索引</title>
<svg:script xmlns:syntaxdiagram-svg="http://www.moldflow.com/2006/syntaxdiagram-svg" xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" type="text/ecmascript" xlink:href="../../js/constants.js"/><svg:script xmlns:syntaxdiagram-svg="http://www.moldflow.com/2006/syntaxdiagram-svg" xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" type="text/ecmascript" xlink:href="../../js/diagram.js"/><svg:script xmlns:syntaxdiagram-svg="http://www.moldflow.com/2006/syntaxdiagram-svg" xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" type="text/ecmascript" xlink:href="../../js/sequence.js"/><svg:script xmlns:syntaxdiagram-svg="http://www.moldflow.com/2006/syntaxdiagram-svg" xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" type="text/ecmascript" xlink:href="../../js/loop.js"/><svg:script xmlns:syntaxdiagram-svg="http://www.moldflow.com/2006/syntaxdiagram-svg" xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" type="text/ecmascript" xlink:href="../../js/decision.js"/><svg:script xmlns:syntaxdiagram-svg="http://www.moldflow.com/2006/syntaxdiagram-svg" xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" type="text/ecmascript" xlink:href="../../js/revdecision.js"/><svg:script xmlns:syntaxdiagram-svg="http://www.moldflow.com/2006/syntaxdiagram-svg" xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" type="text/ecmascript" xlink:href="../../js/boxed.js"/><svg:script xmlns:syntaxdiagram-svg="http://www.moldflow.com/2006/syntaxdiagram-svg" xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" type="text/ecmascript" xlink:href="../../js/unboxed.js"/><svg:script xmlns:syntaxdiagram-svg="http://www.moldflow.com/2006/syntaxdiagram-svg" xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" type="text/ecmascript" xlink:href="../../js/text.js"/><svg:script xmlns:syntaxdiagram-svg="http://www.moldflow.com/2006/syntaxdiagram-svg" xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" type="text/ecmascript" xlink:href="../../js/void.js"/><svg:script xmlns:syntaxdiagram-svg="http://www.moldflow.com/2006/syntaxdiagram-svg" xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" type="text/ecmascript" xlink:href="../../js/notecontainer.js"/><svg:script xmlns:syntaxdiagram-svg="http://www.moldflow.com/2006/syntaxdiagram-svg" xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" type="text/ecmascript" xlink:href="../../js/note.js"/><svg:script xmlns:syntaxdiagram-svg="http://www.moldflow.com/2006/syntaxdiagram-svg" xmlns="http://www.w3.org/1999/xhtml" xmlns:svg="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" type="text/ecmascript" xlink:href="../../js/main.js"/><link rel="stylesheet" type="text/css" href="css/syntaxdiagram.css"/></head>
<body id="chapter01">


  <h1 class="title topictitle1">如何创建索引</h1>

  
  <div class="body conbody">
    <p class="p">全文检索的索引创建过程一般有以下几步。</p>

    <div class="section"><h2 class="title sectiontitle">原文档【Document】</h2>
      
      <p class="p">为了方便说明索引创建过程，这里特意用两个文件为例：</p>

      <p class="p">文件一：Students should be allowed to go out with their friends, but not allowed to drink beer.</p>

      <p class="p">文件二：My friend Jerry went to school to see his students but found them drunk which is not
allowed.</p>

    </div>

    <div class="section"><h2 class="title sectiontitle">分词【Tokenizer】</h2>
      
      <p class="p">分词组件(Tokenizer)会做以下几件事情(此过程称为Tokenize)：</p>

      <div class="p"><ol class="ol">
          <li class="li">将文档分成一个一个单独的单词。</li>

          <li class="li">去除标点符号。</li>

          <li class="li">去除停词(Stop word)。</li>

        </ol>
</div>

      <p class="p">所谓停词(Stop word)就是一种语言中最普通的一些单词，由于没有特别的意义，因而大多数情况下不能成为搜索的关键词，因而创建索引时，这种词会被去掉而减少索引的大小。</p>

      <p class="p">英语中停词(Stop word)如：“the”,“a”，“this”等。
</p>

      <p class="p">对于每一种语言的分词组件(Tokenizer)，都有一个停词(stop word)集合。
</p>

      <p class="p">经过分词(Tokenizer)后得到的结果称为词元(Token)。
</p>

      <p class="p">在我们的例子中，便得到以下词元(Token)：“Students”，“allowed”，“go”，“their”，“friends”，“allowed”，“drink”，“beer”，“My”，“friend”，“Jerry”，“went”，“school”，“see”，“his”，“students”，“found”，“them”，“drunk”，“allowed”。</p>

    </div>

    <div class="section"><h2 class="title sectiontitle">语言处理 【Linguistic Processor】</h2>
      
      <p class="p">语言处理组件(linguistic processor)主要是对得到的词元(Token)做一些同语言相关的处理。</p>

      <p class="p">对于英语，语言处理组件(Linguistic Processor)一般做以下几点：</p>

      <div class="p"><ol class="ol">
          <li class="li">变为小写(Lowercase)。</li>

          <li class="li">将单词缩减为词根形式，如“cars”到“car”等。这种操作称为：stemming。</li>

          <li class="li">将单词转变为词根形式，如“drove”到“drive”等。这种操作称为：lemmatization。</li>

        </ol>
</div>

      <p class="p">Stemming 和 lemmatization的异同：</p>

      <div class="p"><ul class="ul">
          <li class="li">相同之处：Stemming 和lemmatization 都要使词汇成为词根形式。</li>

          <li class="li">两者的方式不同：<ol class="ol">
              <li class="li">Stemming 采用的是“缩减”的方式：“cars”到“car”，“driving”到“drive”。</li>

              <li class="li">Lemmatization 采用的是“转变”的方式：“drove”到“drove”，“driving”到“drive”。</li>

            </ol>
</li>

          <li class="li">两者的算法不同：<ol class="ol">
              <li class="li">Stemming 主要是采取某种固定的算法来做这种缩减，如去除“s”，去除“ing”加“e”，将“ational”变为“ate”，将“tional”变为“tion”。</li>

              <li class="li">Lemmatization 主要是采用保存某种字典的方式做这种转变。比如字典中有“driving”到“drive”，“drove”到“drive”，“am, is, are”到“be”的映射，做转变时，只要查字典就可以了。</li>

            </ol>
</li>

          <li class="li">Stemming 和lemmatization 不是互斥关系，是有交集的，有的词利用这两种方式都能达到相同的转换。</li>

        </ul>
</div>

      <p class="p">语言处理组件(linguistic processor)的结果称为词(Term)。</p>

      <p class="p">在我们的例子中，经过语言处理，得到的词(Term)如下：</p>

      <p class="p">“student”，“allow”，“go”，“their”，“friend”，“allow”，“drink”，“beer”，“my”，“friend”，“jerry”，“go”，“school”，“see”，“his”，“student”，“find”，“them”，“drink”，“allow”。</p>

      <p class="p">也正是因为有语言处理的步骤，才能使搜索drove，而drive 也能被搜索出来。</p>

    </div>

    <div class="section"><h2 class="title sectiontitle">将得到的词【Term】传给索引组件【Indexer】</h2>
      
      <p class="p">索引组件(Indexer)主要做以下几件事情：</p>

      <div class="p"><ol class="ol">
          <li class="li">
            <p class="p">利用得到的词(Term)创建一个字典。</p>

            <p class="p">在我们的例子中字典如下：</p>

            <div class="p">
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" class="table" frame="border" border="1" rules="all"><caption><span class="tablecap">表 1. 字典表</span></caption>
                  
                  
                  <thead class="thead" align="left">
                    <tr class="row">
                      <th class="entry" valign="top" id="d510e222">Term</th>

                      <th class="entry" valign="top" id="d510e225">Document ID</th>

                    </tr>

                  </thead>

                  <tbody class="tbody">
                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">student</td>

                      <td class="entry" valign="top" headers="d510e225 ">1</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">allow</td>

                      <td class="entry" valign="top" headers="d510e225 ">1</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">go</td>

                      <td class="entry" valign="top" headers="d510e225 ">1</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">their</td>

                      <td class="entry" valign="top" headers="d510e225 ">1</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">friend</td>

                      <td class="entry" valign="top" headers="d510e225 ">1</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">allow</td>

                      <td class="entry" valign="top" headers="d510e225 ">1</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">drink</td>

                      <td class="entry" valign="top" headers="d510e225 ">1</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">beer</td>

                      <td class="entry" valign="top" headers="d510e225 ">1</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">my</td>

                      <td class="entry" valign="top" headers="d510e225 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">friend</td>

                      <td class="entry" valign="top" headers="d510e225 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">jerry</td>

                      <td class="entry" valign="top" headers="d510e225 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">go</td>

                      <td class="entry" valign="top" headers="d510e225 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">school</td>

                      <td class="entry" valign="top" headers="d510e225 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">see</td>

                      <td class="entry" valign="top" headers="d510e225 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">his</td>

                      <td class="entry" valign="top" headers="d510e225 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">his</td>

                      <td class="entry" valign="top" headers="d510e225 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">find</td>

                      <td class="entry" valign="top" headers="d510e225 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">them</td>

                      <td class="entry" valign="top" headers="d510e225 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">drink</td>

                      <td class="entry" valign="top" headers="d510e225 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e222 ">allow</td>

                      <td class="entry" valign="top" headers="d510e225 ">2</td>

                    </tr>

                  </tbody>

                </table>
</div>
</div>

          </li>

          <li class="li">
            <p class="p">对字典按字母顺序进行排序。</p>

            <div class="p">
<div class="tablenoborder"><table cellpadding="4" cellspacing="0" summary="" class="table" frame="border" border="1" rules="all"><caption><span class="tablecap">表 2. </span></caption>
                  
                  
                  <thead class="thead" align="left">
                    <tr class="row">
                      <th class="entry" valign="top" id="d510e439">Term</th>

                      <th class="entry" valign="top" id="d510e442">Document ID</th>

                    </tr>

                  </thead>

                  <tbody class="tbody">
                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">allow</td>

                      <td class="entry" valign="top" headers="d510e442 ">1</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">allow</td>

                      <td class="entry" valign="top" headers="d510e442 ">1</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">allow</td>

                      <td class="entry" valign="top" headers="d510e442 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">beer</td>

                      <td class="entry" valign="top" headers="d510e442 ">1</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">drink</td>

                      <td class="entry" valign="top" headers="d510e442 ">1</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">drink</td>

                      <td class="entry" valign="top" headers="d510e442 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">find</td>

                      <td class="entry" valign="top" headers="d510e442 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">friend</td>

                      <td class="entry" valign="top" headers="d510e442 ">1</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">friend</td>

                      <td class="entry" valign="top" headers="d510e442 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">go</td>

                      <td class="entry" valign="top" headers="d510e442 ">1</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">go</td>

                      <td class="entry" valign="top" headers="d510e442 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">his</td>

                      <td class="entry" valign="top" headers="d510e442 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">jerry</td>

                      <td class="entry" valign="top" headers="d510e442 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">my</td>

                      <td class="entry" valign="top" headers="d510e442 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">school</td>

                      <td class="entry" valign="top" headers="d510e442 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">see</td>

                      <td class="entry" valign="top" headers="d510e442 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">student</td>

                      <td class="entry" valign="top" headers="d510e442 ">1</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">student</td>

                      <td class="entry" valign="top" headers="d510e442 ">2</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">their</td>

                      <td class="entry" valign="top" headers="d510e442 ">1</td>

                    </tr>

                    <tr class="row">
                      <td class="entry" valign="top" headers="d510e439 ">them</td>

                      <td class="entry" valign="top" headers="d510e442 ">2</td>

                    </tr>

                  </tbody>

                </table>
</div>
</div>

          </li>

          <li class="li">
            <p class="p">合并相同的词(Term)成为文档倒排(Posting List)链表</p>

            <p class="p"><img class="image" src="../../images/chapter01_02_01.png" alt="倒排序链表"/></p>

            <p class="p">在此表中，有几个定义：</p>

            <div class="p"><ul class="ul">
                <li class="li">Document Frequency 即文档频次，表示总共有多少文件包含此词(Term)。</li>

                <li class="li">Frequency 即词频率，表示此文件中包含了几个此词(Term)。</li>

              </ul>
</div>

            <p class="p">所以对词(Term) “allow”来讲，总共有两篇文档包含此词(Term)，从而词(Term)后面的文档链表总共有两项，第一项表示包含“allow”的第一篇文档，即1 号文档，此文档中，“allow”出现了2 次，第二项表示包含“allow”的第二个文档，是2 号文档，此文档中，“allow”出现了1次。</p>

            <p class="p">到此为止，索引已经创建好了，我们可以通过它很快的找到我们想要的文档。</p>

            <p class="p">而且在此过程中，我们惊喜地发现，搜索“drive”，“driving”，“drove”，“driven”也能够被搜到。</p>

            <p class="p">因为在我们的索引中，“driving”，“drove”，“driven”都会经过语言处理而变成“drive”，在搜索时，如果您输入“driving”，输入的查询语句同样经过我们这里的一到三步，从而变为查询“drive”，从而可以搜索到想要的文档。</p>

          </li>

        </ol>
</div>

    </div>

    <div class="section">
      <a class="xref" href="http://www.ossez.com/forum.php?mod=viewthread&amp;tid=18378&amp;fromuid=426" target="_blank">http://www.ossez.com/forum.php?mod=viewthread&amp;tid=18378&amp;fromuid=426</a>
    </div>

  </div>

<div class="related-links">
<div class="familylinks">
<div class="parentlink"><strong>父主题：</strong> <a class="link" href="../../tasks/chapter01/chapter01.html">基本原理</a></div>
</div>
</div>

</body>
</html>